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Why  GAO  Did  This  Study 

Cost  and  schedule  growth  in  DOD 
major  defense  acquisition  programs 
persist,  and  some  acquisition  reform 
proponents  believe  such  growth  is  due 
to  unplanned  changes  in  program 
requirements  (commonly  referred  to  as 
"requirements  creep").  GAO  found  in 
June  2015  that  cost  and  schedule 
growth  are  often  more  directly  related 
to  a  lack  of  systems  engineering, 
which,  if  done,  would  reduce  risk  by 
introducing  discipline  and  rigor  into  the 
process  of  defining  and  understanding 
a  program's  initial  requirements. 

House  Armed  Services  Committee 
Report  114-102  contained  a  provision 
for  GAO  to  review  the  DOD 
requirements  process.  This  report  (1) 
identifies  a  framework  for  assessing 
the  challenge  posed  by  weapon 
system  requirements  and  the  extent  of 
systems  engineering  done  before 
product  development  begins;  (2) 
illustrates  the  relationship  between 
systems  engineering  and  program 
outcomes;  and  (3)  assesses 
implications  for  program  oversight. 
GAO  analyzed  a  non-generalizable 
sample  of  nine  case  studies.  GAO 
assessed  the  extent  to  which  systems 
engineering  was  conducted  before 
development  by  reviewing  program 
requirements  and  analyzing  cost  and 
schedule  documentation  for  each  case 
study.  GAO  also  reviewed  prior  GAO 
work  and  interviewed  DOD  officials. 

What  GAO  Recommends 

To  support  oversight  and  inform 
budget  decisions,  Congress  should 
consider  requiring  DOD  to  report  on 
systems  engineering  status  of  each 
major  acquisition  program  in  the 
department’s  annual  budget  request. 


View  GAO-17-77.  For  more  information, 
contact  Michael  J.  Sullivan  at  (202)  512-4841 
or  sullivanm@gao.gov. 
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What  GAO  Found 

GAO’s  analysis  of  nine  case  studies  identified  four  factors  that  frame  the 
challenge  posed  by  a  given  weapon  system’s  requirements:  acquisition 
approach,  technology  status,  design  maturity,  and  system  interdependency. 
Systems  engineering  is  the  primary  means  for  determining  whether  and  how  that 
challenge  can  be  met.  It  is  a  disciplined  learning  process  that  translates 
requirements  into  specific  design  features  and  thus  identifies  key  risks  to  be 
resolved.  GAO’s  prior  best  practices  work  has  found  that  if  detailed  systems 
engineering  is  done  early,  a  program  can  resolve  such  risks  through  trade-offs 
and  additional  investments  before  a  program  starts.  A  key  point  in  systems 
engineering  where  this  match  can  be  assessed  is  the  preliminary  design.  As 
shown  below,  establishing  a  preliminary  design  through  early  detailed  systems 
engineering  portends  better  program  outcomes  than  doing  so  after  program  start. 


Timing  of  Systems  Engineering  for  Problematic  and  Successful  Programs 
Product  development  start 


Source:  GAO  analysis  of  Department  of  Defense  guidance  and  selected  program  data.  |  GAO-17-77 

GAO’s  analysis  of  selected  Department  of  Defense  (DOD)  programs  illustrates 
the  relationship  among  the  four  factors,  systems  engineering,  and  program 
outcomes.  Programs  with  modest  requirements  and  early  detailed  systems 
engineering  had  better  outcomes.  For  example,  the  Small  Diameter  Bomb 
Increment  I  program,  which  delivered  within  cost  and  schedule  estimates,  had  an 
incremental  approach,  mature  technologies,  a  derivative  design,  and  detailed 
systems  engineering  before  development  began.  Programs  that  began  with  more 
challenging  requirements  and  insufficient  systems  engineering  reported  worse 
outcomes.  For  example,  the  F-35  Lightning  II,  which  has  encountered  significant 
cost  and  schedule  problems,  began  development  with  a  single-step  approach,  a 
highly  complex  design,  immature  technologies,  and  little  systems  engineering. 

Understanding  the  dynamic  between  a  program’s  requirements,  risks,  and  the 
requisite  systems  engineering  effort  has  important  implications  for  oversight.  A 
particular  challenge  is  that  Congress  often  must  consider  requests  to  authorize 
and  fund  a  new  program  in  advance  of  the  start  of  product  development,  when 
the  business  case  would  be  better  established.  DOD  policy  requires  that  DOD 
decision  makers  have  information  about  a  proposed  program’s  risk  factors  and 
systems  engineering  status,  in  a  systems  engineering  plan,  at  the  start  of  a  new 
program.  However,  it  is  not  clear  whether  Congress  also  has  this  information  at 
that  time.  A  systems  engineering  plan  could  provide  more  robust  information  to 
Congress  when  considering  a  budget  request  to  start  a  new  program.  In 
commenting  on  a  draft  of  this  report  DOD  disagreed. 
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U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


November  17,  2016 
Congressional  Committees 

The  Department  of  Defense  (DOD)  expects  to  have  invested  a  total  of 
more  than  $1.4  trillion  in  development  and  procurement  of  the  major 
defense  acquisition  programs  in  its  current  portfolio.  This  portfolio  has 
experienced  cost  growth  of  48  percent  since  first  full  estimates  and 
average  delays  in  delivering  initial  capabilities  of  almost  30  months.  Some 
acquisition  reform  proponents  believe  that  unplanned  changes  in  program 
requirements — ’’requirements  creep” — after  the  Joint  Requirements 
Oversight  Council1  has  approved  the  requirements  and  handed  them  off 
to  the  acquisition  community  are  a  primary  cause  of  these  ongoing  cost 
and  schedule  problems  in  many  programs.  In  June  2015,  we  found  that 
cost  and  schedule  growth  in  major  acquisition  programs  were  not 
necessarily  a  direct  result  of  requirements  changes,  but  were  instead 
more  directly  related  to  a  lack  of  discipline  and  rigor  in  the  process  of 
defining  and  understanding  a  program’s  initial  requirements.2 

House  Armed  Services  Committee  Report  1 14-102  contained  a  provision 
for  us  to  review  the  DOD  requirements  process.3  This  report  (1)  identifies 
a  framework  for  assessing  the  challenges  and  risks  posed  by 
requirements  and  the  extent  of  systems  engineering  done  before  product 
development  begins;  (2)  illustrates  the  relationship  between  systems 
engineering  and  program  outcomes;  and  (3)  assesses  implications  for 
program  oversight. 

To  conduct  our  work,  we  selected  a  non-generalizable  sample  of  nine 
major  defense  acquisition  programs,  made  up  of  various  types  of  systems 
from  each  military  service,  to  conduct  case  study  assessments.  The 
specific  case  study  programs  we  selected  are  identified  in  table  1. 


1  The  Joint  Requirements  Oversight  Council,  which  is  chaired  by  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff  and  is  comprised  of  the  Vice  Chiefs  of  Staff  of  each  military  service 
and  the  Combatant  Commanders,  has  the  mission  and  responsibility  to  identify,  assess, 
and  approve  joint  military  requirements  including  existing  systems  and  equipment  to  meet 
the  National  Military  Strategy. 

2  GAO,  Defense  Acquisition  Process:  Military  Service  Chiefs’  Concerns  Reflect  Need  to 
Better  Define  Requirements  before  Programs  Start,  GAO-15-469  (Washington,  D.C.:  June 
11,2015). 

3H.R.  Rep.  No.  114-102,  at  184-185  (May  5,  2015) 
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Table  1:  Case  Study  Programs 

Program 

Description 

KC-46A  Tanker  Modernization  Program 

Aerial  refueling  tanker  aircraft 

Global  Positioning  System  III 

Satellites 

Small  Diameter  Bomb  Increment  1 

Air-to-ground  precision  munitions 

Integrated  Air  and  Missile  Defense 

Network  of  sensors  and  weapons  with  a 
common  battle  command  system 

Paladin  Integrated  Management 

Self-propelled  howitzer  and  tracked 
ammunition  carrier 

F-35  Lightning  II  Program 

Stealthy,  strike  fighter  aircraft 

Joint  Light  Tactical  Vehicle 

Tactical  wheeled  vehicles 

CH-53K  Heavy  Lift  Replacement  Helicopter 

Personnel  and  equipment  transport  aircraft 

P-8A  Poseidon  Multi-mission  Maritime 
Aircraft  Increment  1 

Anti-surface,  anti-submarine  and 
intelligence,  surveillance  and 
reconnaissance  aircraft 

Source:  GAO  analysis  of  DOD  data.  |  GAO-17-77 


To  assess  the  challenges  and  risks  associated  with  program 
requirements  and  the  extent  of  the  systems  engineering  done  by  each  of 
the  nine  programs  before  development  began,  we  analyzed  top-level 
program  requirements  such  as  key  performance  parameters,  key  system 
attributes,  and  other  system  attributes,  as  well  as  lower-level,  derived 
requirements  like  system  design  specifications  and  design  drawings.  In 
addition,  we  assessed  documents  and  data  from  program  system 
engineering  reviews  and  reported  technology  readiness  levels,  and 
reviewed  program  acquisition  strategies,  acquisition  baselines,  and 
selected  acquisition  reports.  We  reviewed  cost,  schedule,  and 
requirements  data  and  interviewed  agency  officials  knowledgeable  about 
the  data.  We  determined  that  the  data  were  sufficiently  reliable  for  the 
purposes  of  our  reporting  objectives.  To  assess  the  relationship  between 
systems  engineering  and  program  outcomes,  we  analyzed  requirements, 
cost,  and  schedule  documentation  for  each  of  our  case  study  programs, 
and  then  met  with  relevant  program  officials  and  prime  contractors  to 
obtain  relevant  program  documents,  data,  and  historical  information.  To 
identify  program  oversight  implications,  we  also  reviewed  relevant 
acquisition  statutes,  DOD  acquisition  policy,  systems  engineering 
guidance,  and  previous  GAO  reports  examining  weapons  systems 
acquisitions  and  best  practices  for  product  development.  To  obtain 
additional  insights  into  these  implications,  we  spoke  with  knowledgeable 
DOD  officials  and  the  prime  contractors  for  our  case  study  programs.  See 
appendix  I  for  additional  information  on  our  objectives,  scope,  and 
methodology. 
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We  conducted  this  performance  audit  from  June  2015  to  November  201 6 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


In  our  past  work  examining  weapon  system  acquisitions  and  best 
practices  for  product  development,  we  have  found  that  leading 
commercial  firms  and  successful  DOD  programs  pursue  an  acquisition 
approach  that  is  anchored  in  knowledge,  whereby  high  levels  of  product 
knowledge  are  demonstrated  at  critical  points  in  the  acquisition  process. 
The  first  key  point  in  this  knowledge-based  process  occurs  when  a  match 
is  achieved  between  the  customer’s  needs  or  requirements  and  available 
resources — including  technology,  design,  time,  and  funding.  That  match 
establishes  a  sound  basis  for  a  program  business  case  and  supports  the 
decision  to  invest  in  product  development.  Achieving  a  match  between 
requirements  and  resources  prior  to  committing  to  a  product  development 
program  reduces  risk  and  uncertainty  and  sets  the  program  up  for 
success.  We  have  previously  found  that  key  enablers  of  a  good  business 
case  include: 

•  Firm,  feasible  requirements:  Requirements  should  be  clearly  defined, 
affordable,  and  clearly  informed — thus  tempered — by  systems 
engineering.  Once  programs  begin,  requirements  should  not  change 
without  assessing  their  potential  disruption  to  the  program. 

•  Mature  technology:  Science  and  technology  organizations  should 
shoulder  the  technology  development  burden,  proving  technologies 
can  work  as  intended  before  they  are  included  in  a  weapon  system 
program.  The  principle  is  not  to  avoid  technical  risk  but  rather  take  risk 
early  and  resolve  it  ahead  of  the  start  of  product  development. 

•  Incremental,  knowledge-based  acquisition  strategy:  Rigorous  systems 
engineering  coupled  with  more  achievable  requirements  are  essential 
to  achieve  faster  delivery  of  needed  capability  to  the  warfighter. 
Building  on  mature  technologies,  such  a  strategy  provides  time, 
money,  and  other  resources  for  a  stable  design,  building  and  testing 
of  prototypes,  and  demonstration  of  mature  production  processes. 

•  Realistic  cost  estimate:  Sound  cost  estimates  depend  on  a 
knowledge-based  acquisition  strategy,  independent  assessments,  and 
sound  methodologies. 
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Major  defense  acquisition  programs  go  through  a  series  of  phases  as 
they  progress  from  the  identification  of  the  top-level  requirements  for  new 
capability,  through  initial  planning  of  a  solution,  to  product  development, 
and  finally  production  and  deployment  of  a  fielded  system.  Within  DOD, 
the  process  of  identifying  and  understanding  requirements  typically 
begins  when  a  sponsor,  usually  a  military  service,  submits  an  Initial 
Capabilities  Document  that  identifies  the  existence  of  a  capability  gap;  the 
operational  risks  associated  with  the  gap;  and  a  recommended  solution  or 
preferred  set  of  solutions  for  filling  the  gap.  Potential  solutions  are  then 
assessed  in  an  Analysis  of  Alternatives,  system  capabilities  are  chosen, 
and  top-level  capability  or  operational  requirements  are  then  defined  in  a 
draft  Capability  Development  Document  that  goes  through  several  stages 
of  service-  and  DOD-level  review  and  validation.  After  the  top-level 
capability  requirements  are  defined,  they  are  then  decomposed  into  more 
specific  capability  requirements,  known  as  the  performance  specification, 
and  then  into  a  series  of  more  detailed  design  requirements. 

Progressively  more  detailed  descriptions  of  the  system  design  are 
established  in  what  systems  engineering  literature  refers  to  as 
configuration  baselines.  Within  DOD,  the  configuration  baselines  are 
called  the  “functional  baseline”  that  details  the  system-level  performance 
requirements;  the  “allocated  baseline”  that  establishes  a  preliminary 
design  and  defines  the  subsystems  and  how  they  are  to  work  together; 
and  the  “product  baseline”  that  describes  the  final  design  and  provides 
component-level  details  (see  figure  1). 
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Figure  1:  Requirements  Decomposition 


Initial  Capability  Document  identifies  capability  gap  in  terms  of  the  functional 
area,  the  relevant  range  of  military  operations,  desired  effects  and  time  with  policy 
implications  and  constraints 


Capability  Development  Document  identifies  operational  requirements  of  the 
system  in  the  form  of  key  performance  parameters  and  key  systems  attributes 


Defines  performance  requirements-based  on  understanding  of  the 
top-level  system  requirements-that  are  typically  put  on  contract  to  support 
further  requirements  analysis  and  design  activities 


Details  performance  specification  for  the  overall  system- Weapons  System 
Specification-  and  the  tests  needed  to  verify  and  validate  overall  system. 
Baseline  established  by  the  System  Functional  Review. 


Defines  the  subsystems  and  how  function  and  performance  requirements 
are  allocated  across  the  subsystems  and  how  they  are  to  work  with  other 
internal  subsystems  and  external  systems  based  on  preliminary  design 
Baseline  established  by  the  Preliminary  Design  Review 


Defines  all  functional  and  physical  characteristics  of  the  components 
that  make  up  the  subsystems  including  build  to  specifications  based  on 
the  final  design  Baseline  established  by  the  Critical  Design  Review 


Source:  GAO  analysis  of  Department  of  Defense  policy  and  guidance.  |  GAO-17-77 


Systems  engineering  is  a  disciplined  process  to  transform  top-level 
capability  requirements  into  detailed,  lower-level  design  requirements  that 
can  be  achieved  with  available  resources.  Programs  are  required  to 
prepare  a  systems  engineering  plan  as  a  management  tool  to  guide 
systems  engineering  activities  over  the  life  of  a  program.  According  to 
DOD,  the  discipline  of  this  process  provides  the  control  and  traceability  to 
develop  solutions  that  meet  customer  needs.  The  systems  engineering 
process  can  be  depicted  in  a  V-shape  that  starts  when  a  capability  gap  is 
identified  and  a  proposed  solution  for  filling  that  gap  is  selected.  Top-level 
capability  requirements  are  then  defined  and  decomposed  into  lower-level 
design  requirements  until  a  final  design  is  established.  As  this  process 
takes  place,  requirements  become  more  specific  and  the  number  of 
requirements  at  each  successive  level  grows.  This  growth  can  be 
exponential,  with  tens  of  thousands  of  detailed  design  requirements 
derived  from  a  relatively  small  number  of  capability  requirements.  Figure 
2  illustrates  the  relationship  between  requirements  decomposition  and 
systems  engineering. 
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Figure  2:  Requirements  and  Systems  Engineering 


Requirements 


Systems  Engineering 


10s 


10,000s. 

Source:  GAO  analysis  of  Department  of  Defense  policy,  guidance  and  selected  programs.  |  GAO-17-77 


Note:  Because  this  review  focused  on  the  requirements  process  that  leads  to  a  final  design,  the 
product,  validated  solution,  and  delivered  capability  portions  of  the  systems  engineering  process  that 
focus  on  developing,  testing,  and  delivering  a  final  product  are  not  discussed. 


As  more  detailed  design  requirements  are  identified,  decision  makers  can 
make  informed  trades  between  the  requirements  and  available  resources, 
potentially  achieving  a  match  and  establishing  a  sound  basis  for  a 
program  business  case.  As  the  requirements  decomposition  process 
takes  place,  engineering  and  design  knowledge  grows  and  risk 
decreases,  leading  to  more  realistic  cost  and  schedule  estimates  and 
more  predictable  program  outcomes. 
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Four  Factors  Frame 
the  Challenge  Posed 
by  Requirements,  and 
Detailed  Systems 
Engineering  Is  Key  to 
Knowing  Whether 
and  How  the 
Challenge  Can  Be 
Met 


Our  analysis  of  nine  selected  DOD  programs,  supported  by  our  previous 
work  examining  best  practices  for  product  development,  identified  four 
key  factors  that  provide  insight  into  the  challenge  posed  by  a  system’s 
top-level  capability  requirements  and  the  related  risk.  While  they  are  not 
the  only  factors  that  could  be  considered,  we  found  that  they  were 
prominent  and  observable  early  in  our  case  study  programs.  Those  four 
factors  are: 

•  Acquisition  approach  -  indicates  whether  a  program  intends  to  take  an 
incremental,  evolutionary  approach  or  a  single-step  approach  to  meet 
all  capability  requirements.  According  to  GAO  best  practices,  an 
incremental  approach  reduces  risk  by  developing  and  delivering  a 
product  in  a  series  of  enhanced  interim  capabilities  until  final 
capability  is  reached.  In  contrast,  a  single-step  approach  increases 
risk  by  attempting  to  deliver  the  final  capability  all  at  once  without  any 
interim  capabilities. 


•  Technology  status  -  indicates  the  extent  to  which  the  critical 
technologies  for  a  proposed  system  are  mature  at  the  start  of  product 
development.4  According  to  GAO  best  practices,  technologies  are 
ready  for  inclusion  in  a  product  development  program  when  they  have 
been  demonstrated  in  a  realistic,  operational  environment  (i.e. , 
Technology  Readiness  Level  (TRL)  7),  proving  that  they  can  work  as 
intended.  Demonstrating  technologies  in  an  operational  environment 
provides  a  higher  level  of  technology  understanding  and  reduces  risk 
prior  to  starting  product  development.  Beginning  system  development 
with  less  mature  technologies  (i.e.,  TRL  lower  than  7)  increases 
program  risk  because  unexpected  technology  challenges  could 
significantly  affect  product  design. 

•  Design  maturity  -  indicates  the  extent  to  which  a  proposed  system’s 
design  is  either  derived  from  an  existing  commercial  or  DOD 
system — whether  operational  or  prototype — or  is  new  and 
unprecedented.  According  to  GAO  best  practices,  designs  derived 
from  existing  systems,  whether  commercial  or  DOD,  by  nature  have 
more  knowledge  available  when  product  development  begins,  thus 


4  DOD  policy  requires  critical  technologies  to  be  assessed  on  a  scale  from  1  to  9  with  9 
being  the  most  mature.  Demonstration  in  a  relevant  environment  is  TRL  6  and 
demonstration  in  an  operational  environment  is  TRL  7.  In  addition,  a  major  defense 
acquisition  program  generally  may  not  receive  approval  for  development  start  until  the 
milestone  decision  authority  certifies  that  the  technology  in  the  program  has  been 
demonstrated  in  a  relevant  environment.  10  U.S.C.  §  2366b. 
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reducing  risk.  In  contrast,  unprecedented  designs  are  by  nature  more 
complex  and  inherently  higher  risk. 

•  System  interdependency  -  indicates  the  extent  to  which  a  system 
relies  on  another  system  or  group  of  systems  being  developed  and 
managed  separately  to  provide  its  required  capabilities.  According  to 
DOD  acquisition  guidance,  although  DOD  programs  should  not  be 
acquired  in  isolation,  a  program  office  developing  a  more  independent 
system  generally  has  more  control  over  requirements,  schedule, 
funding,  and  interfaces,  among  other  factors.  In  contrast,  a  program 
office  developing  a  system  that  is  more  dependent  on  other 
systems — like  a  system  of  systems — generally  has  less  control,  thus 
making  its  requirements  more  challenging  to  achieve  and  increasing 
risk.5  For  example,  systems  of  systems  tend  to  be  more  complex  and 
must  manage  (1)  interfaces  with  other  systems  that  have  potentially 
unaligned  performance  requirements,  (2)  the  need  for  collaborative 
governance  with  separately  managed  programs,  and  (3)  differences  in 
program  phases  of  development. 

As  we  discuss  in  the  following  pages,  the  case  study  programs  with  top- 
level  capability  requirements  that  posed  a  more  modest  challenge  took  an 
incremental  acquisition  approach,  incorporated  more  mature 
technologies,  used  a  design  that  was  derived  from  an  existing  product  or 
prototype,  and  limited  dependence  on  systems  being  developed  and 
managed  separately,  reducing  the  amount  of  risk  being  carried  forward 
into  product  development.  In  contrast,  the  programs  with  top-level 
capability  requirements  that  were  more  challenging  generally  assumed 
full  capability  would  be  delivered  in  a  single-step,  incorporated  immature 
critical  technologies,  used  a  new  and  unprecedented  system  design,  and 
had  full  capability  that  in  some  cases  was  critically  dependent  on 
separately  developed  and  managed  systems  without  sufficiently 
mitigating  risk  before  starting  product  development. 

The  fact  that  some  requirements  can  be  more  challenging  than  others 
does  not  necessarily  mean  that  problems  are  unavoidable.  Rather,  the 
ensuing  systems  engineering  effort,  if  timely  and  sufficiently  vigorous,  can 
provide  the  needed  confidence  that  the  requirements  are  achievable.  Our 
previous  work  in  best  practices  has  found  that  conducting  detailed 
systems  engineering  analysis  before  starting  product  development 
contributes  to  understanding  the  requirements  challenge  and  identifying 


5  Department  of  Defense  Acquisition  Guidebook,  Ch.  4,  para.  4.1.3. 
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and  mitigating  associated  risks.6  Systems  engineering  is  the  primary 
means  for  determining  whether  and  how  the  challenge  posed  by  a 
program’s  requirements  can  be  met  with  available  resources.  It  is  a 
disciplined  learning  process  that  translates  capability  requirements  into 
specific  design  features  and  thus  identifies  key  risks  to  be  resolved.  Our 
prior  best  practices  work  has  indicated  that  if  detailed  systems 
engineering  is  done  before  the  start  of  product  development,  the  program 
can  resolve  these  risks  through  trade-offs  and  additional  investments, 
ensuring  that  risks  have  been  sufficiently  retired  or  that  they  are  clearly 
understood  and  adequately  resourced  if  they  are  being  carried  forward. 
Figure  3  depicts  this  relationship  between  the  timing  of  systems 
engineering  and  the  start  of  product  development  for  successful  and 
problematic  programs. 


Figure  3:  Timing  of  Systems  Engineering  for  Successful  and  Problematic  Programs 


Product 

development  start 
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Detailed  systems  engineering  (final  design) 

Allocated  baseline 
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*  (final  design) 

Successful 

programs 

Detailed  systems  engineering 

Source:  GAO  analysis  of  Department  of  Defense  guidance  and  selected  program  data.  |  GAO-17-77 


In  our  case-study  analysis,  we  found  that  systems  engineering  is  a 
natural  learning  process  that  takes  place  at  some  point  in  all  programs. 


6  GAO,  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  under  DOD’s  Revised  Policy ,  GAO-06-368  (Washington,  D.C.:  Apr. 
13,  2006);  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early 
Improves  Acquisition  Outcomes.  GAO-02-701  (Washington,  D.C.:  July  15,  2002);  Best 
Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better  Weapon  System 
Outcomes,  GAO-01-288  (Washington,  D.C.:  Mar.  8,  2001). 
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Our  work  in  commercial  best  practices  has  found  that,  at  a  minimum, 
programs  need  to  conduct  systems  engineering  to  decompose  capability 
requirements  into  an  allocated  baseline — that  is,  establish  a  preliminary 
design — before  beginning  a  product  development  program.7  When  the 
bulk  of  the  detailed  systems  engineering  is  done  after  starting  a  product 
development  program  it  often  results  in  cost  and  schedule  growth  and 
creates  inefficiency.  Figure  4  compares  the  points  in  the  requirements 
and  systems  engineering  processes  at  which  problematic  and  successful 
programs  would  begin  product  development. 


Figure  4:  Requirements  and  Systems  Engineering  for  Problematic  and  Successful  Product  Development  Start 
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Source:  GAO  analysis  of  Department  of  Defense  policy,  guidance  and  selected  programs  |  GAO-17-77 


Within  the  current  defense  acquisition  framework,  DOD  decision  makers 
should  assess  a  program’s  risk  factors  and  systems  engineering  progress 
well  in  advance  of  the  start  of  product  development.  For  example,  DOD’s 
acquisition  policy  states  that  program  requirements  must  be  firm  and 


7  GAO-06-368,  GAO-02-701,  GAO-01-288. 
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clearly  stated  and  that  the  risk  of  committing  to  development  is  or  will  be 
adequately  reduced  prior  to  awarding  a  development  contract.  The  policy 
also  emphasizes  the  value  in  establishing  an  allocated  baseline — i.e. 
preliminary  design — prior  to  the  start  of  product  development,  which  is  a 
best  practice  that  contributed  to  good  outcomes  in  our  case  study 
programs.  This  practice  was  also  emphasized  by  Congress  in  the 
Weapon  System  Acquisition  Reform  Act  of  2009,  which  required 
programs  to  hold  a  preliminary  design  review  or  obtain  a  waiver  prior  to 
starting  a  development  program.8 

We  have  historically  found  that  many  of  DOD’s  major  weapon  system 
development  programs  tended  to  conduct  the  bulk  of  systems 
engineering  after  a  development  contract  had  been  awarded  and  product 
development  had  begun.9  In  many  cases,  the  top-level  capability 
requirements  established  by  the  government  at  that  time  still  needed  to 
be  analyzed  by  the  contractor  and  decomposed  into  a  functional  baseline 
(system-level  design),  an  allocated  baseline  (preliminary  design),  and  a 
product  baseline  (final  design).  As  a  result,  those  programs  had  started 
product  development  with  a  limited  understanding  of  the  challenge  posed 
by  requirements. 


8  Pub.  L.  No.  1 11-23,  §  205(a)  (May  22,  2009). 

9  GAO,  Defense  Acquisition  Process:  Military  Service  Chiefs’  Concerns  Reflect  Need  to 
Better  Define  Requirements  before  Programs  Start,  GAO-15-469  (Washington,  D.C.:  June 
1 1 , 201 5);  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs, 
GAO-12-400SP  (Washington,  D.C.:  Mar.  29,  2012);  Defense  Acquisitions:  Assessments 
of  Selected  Weapon  Programs,  GAO-09-326SP  (Washington,  D.C.:  Mar.  30,  2009);  and 
Best  Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better  Weapon 
System  Outcomes,  GAO-01-288  (Washington,  D.C.:  Mar.  8,  2001). 
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Case  Studies 
Illustrate  the 
Relationship  between 
Requirements 
Challenge,  the  Timing 
of  Systems 
Engineering,  and 
Program  Outcomes 


Our  case  studies  illustrate  the  relationship  among  program  outcomes,  the 
four  key  factors  that  frame  the  challenge  posed  by  requirements,  and 
timely  systems  engineering.  Our  analysis  of  development  cost  and 
schedule  data  for  the  nine  case  studies  shows  that,  as  of  December 
201 5,  about  half  of  the  programs  had  reported  relatively  good 
development  cost  and  schedule  outcomes,  while  the  others  had 
experienced  significant  cost  and  schedule  growth  (see  table  2).  Of  the 
programs  we  studied,  the  three  that  began  development  with  more 
modest  requirements  and  had  conducted  detailed  systems  engineering 
generally  had  good  outcomes.  The  three  programs  with  some 
requirements  challenges  that  conducted  systems  engineering  analysis  to 
mitigate  associated  risks  experienced  moderate  cost  and  schedule 
outcomes.  Finally,  the  three  programs  in  our  sample  that  began 
development  with  challenging  requirements  and  had  done  little  systems 
engineering  generally  reported  poor  outcomes. 


Table  2:  Development  Cost  and  Schedule  Outcomes  of  Nine  Case  Study  Programs 
(then-year  dollars  in  millions) 

Program 

Initial 

estimate 

Current 

estimate 

Percent 

change 

Acquisition  cycle  time 
growth  since  initial 
estimates  (in  months) 

KC-46A  Tanker 

Modernization  Program 

$7,149.6 

$6,259.6 

-12% 

14 

Joint  Light  Tactical 

Vehicle 

$1,009.8 

$948.9 

-6% 

19 

Small  Diameter  Bomb 
Increment  1 

$381.3 

$367.7 

-4% 

-1 

Paladin  Integrated 
Management/Ml  09A7 

Family  of  Vehicles 

$1,041.7 

$1,098.6 

5% 

2 

P-8A  Poseidon  Multi¬ 
mission  Maritime  Aircraft 
Increment  1 

$6,975.5 

$7,940.4 

14% 

4 

Global  Positioning 

System  III 

$2,512.0 

$3,018.6 

20% 

n/a 

CH-53K  Heavy  Lift 
Replacement  Helicopter 

$4,366.4 

$6,598.3 

51% 

51 

F-35  Lightning  II  Program 

$34,400.0 

$55,133.0 

60% 

62 

Integrated  Air  and  Missile 
Defense 

$1,672.5 

$2,632.9 

62% 

22 

Source:  GAO  analysis  of  DOD  data.  |  GAO-17-77 

Note:  Acquisition  cycle  time  is  calculated  from  the  start  of  product  development  to  initial  operational 
capability.  We  could  not  calculate  acquisition  cycle  times  for  the  first  increment  of  the  Global 
Positioning  System  III  program  because  initial  operational  capability  will  not  occur  until  satellites  from 
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a  future  increment  are  fielded.  For  the  P-8A  Increment  I  current  estimate,  we  used  the  P-8A  budget 
estimate  from  February  2016  to  separate  increment  I  cost  from  increment  II. 


These  results  are  consistent  with  our  recent  analysis  of  DOD’s  overall 
major  defense  acquisition  portfolio.  In  March  2016,  we  reported  that  while 
program  outcomes  across  the  portfolio  remained  mixed,  DOD’s  newer 
programs  were  reporting  better  cost  outcomes.10  We  found  that  some  of 
those  programs  had  taken  steps  to  retire  risks  through  systems 
engineering  before  starting  product  development  and  thus  began  with  a 
clearer  understanding  of  the  challenge  posed  by  their  requirements. 


Three  More  Successful 
Programs  Had  Modest 
Requirements  and 
Conducted  Detailed 
Systems  Engineering 
Analysis  before 
Development  Start 


The  KC-46A  Tanker  Modernization  (KC-46),  Small  Diameter  Bomb 
Increment  I  (SDB  I),  and  Joint  Light  Tactical  Vehicle  (JLTV)  programs 
provide  examples  in  which  top-level  requirements  generally  posed 
moderate  challenges,  steps  were  taken  to  identify  and  retire  risks  through 
detailed  systems  engineering,  which  enabled  development  of  sound 
business  cases  before  product  development  began.  In  these  cases,  the 
government  and  the  prime  contractor  conducted  systems  engineering  to 
decompose  the  requirements  and  identify  an  allocated  baseline  by  the 
time  they  started  product  development.  As  a  result,  the  programs 
established  cost  and  schedule  baselines  that  were  well  informed, 
contributing  to  relatively  good  outcomes. 


i  n 

GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs. 
GAO-1 6-329SP  (Washington,  D.C.:  Mar.  31, 2016). 
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KC-46  Tanker  Modernization 
Program 


Our  analysis  of  the  KC-46  requirements  challenge  and  the  ensuing 
systems  engineering  effort  is  summarized  in  figure  5. 


Figure  5:  KC-46  Tanker  Modernization  Program 
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Source:  GAO  analysis  of  Department  of  Defense  data-  |  GAO-17-77 
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Officials  from  the  Air  Force  and  the  KC-46  prime  contractor  told  us  that  at 
the  start  of  product  development  in  201 1 ,  they  had  conducted  detailed 
systems  engineering  and  the  system’s  critical  technologies  were  nearing 
maturity.  The  KC-46  was  designed  to  improve  on  the  KC-135’s  refueling 
capacity,  efficiency,  and  capabilities  for  cargo  and  aeromedical 
evacuation,  and  to  integrate  defensive  systems.  The  Air  Force  planned  an 
incremental  acquisition  approach  focused  on  integrating  mature  military 
technologies  onto  a  commercial  aircraft  derivative  design.  Although  the 
government  did  not  hold  a  preliminary  design  review  until  after 
development  began,  prime  contractor  officials  noted  that  they  had 
identified  the  allocated  baseline  as  part  of  their  internal  risk  reduction 
efforts  before  the  development  contract  was  awarded.  As  a  result  the  KC- 
46  program  took  steps  that  enabled  a  sound  business  case.  Because  KC- 
46  development  was  considered  by  the  government  to  be  a  relatively  low- 
risk  effort,  the  prime  contractor  was  awarded  a  fixed  price  incentive 
contract  at  the  start  of  product  development,  which  mitigated  risk  to  the 
government.11  The  prime  contractor  subsequently  discovered  problems 
with  the  aircraft  wiring  and  aerial  refueling  systems  and  encountered  a 
fuel  contamination  incident,  all  of  which  contributed  to  a  14-month  delay 


11  The  KC-46  fixed-price  incentive  contract  for  product  development  limits  the 
government’s  financial  liability  and  provides  the  contractor  incentives  to  reduce  costs  in 
order  to  earn  more  profit.  Barring  any  changes  to  KC-46  requirements  by  the  Air  Force, 
the  contract  specifies  a  target  price  of  $4.4  billion  and  a  ceiling  price  of  $4.9  billion,  at 
which  point  Boeing  must  assume  responsibility  for  all  additional  costs. 
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Small  Diameter  Bomb 
Increment  I 


in  the  delivery  of  initial  operating  capability.  However,  the  government’s 
development  costs  have  decreased  by  12  percent  largely  because  the 
contract  is  fixed  price  and  the  government’s  initial  cost  estimate  assumed 
a  greater  number  of  engineering  changes  than  have  actually  occurred, 
indicating  that  the  Air  Force  understood  that  development  challenges 
remained  and  reflected  that  understanding  in  its  initial  cost  estimates. 

Our  analysis  of  the  SDB  I  requirements  challenge  and  the  ensuing 
systems  engineering  effort  is  summarized  in  figure  6. 


Figure  6:  Small  Diameter  Bomb  Increment  I 
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Source:  GAO  analysis  of  Department  of  Defense  data.  |  GAO-17-77 


At  the  start  of  product  development  in  2003,  the  Air  Force  and  the  SDB  I 
prime  contractor  had  performed  detailed  systems  engineering  to 
understand  the  challenge  posed  by  requirements  and  remaining  risks. 

The  SDB  was  designed  to  meet  a  pressing  Air  Force  need  for  a  low- 
collateral  damage  weapon  small  enough  to  maximize  the  number  that  can 
be  carried  aboard  a  single  aircraft.  The  Air  Force  planned  an  incremental 
acquisition  approach  to  the  overall  SDB  program,  with  SDB  I — the  first 
increment — using  mature  technology  and  based  on  a  design  derived  from 
competitive  prototypes  developed  and  tested  by  the  prime  contractor. 
According  to  program  officials,  all  of  the  system’s  critical  technologies 
were  mature  and  they  had  production  representative  hardware  when 
product  development  started.  The  Air  Force  and  competing  contractors 
conducted  systems  engineering  over  the  course  of  2  years  prior  to 
development  to  refine  requirements  in  light  of  available  resources.  During 
that  time  live  fire  testing  was  conducted  and  the  Air  Force  held 
preliminary  and  critical  design  reviews — key  systems  engineering 
events — with  each  contractor.  According  to  prime  contractor  officials,  their 
interactions  with  the  Air  Force  prior  to  the  start  of  development  allowed 
them  to  get  a  clear  understanding  of  design  requirements.  As  a  result, 
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Joint  Light  Tactical  Vehicle 


they  were  able  to  achieve  a  stable  design  and  provide  a  production 
representative  system  at  the  start  of  product  development.  With  all  of  the 
detailed  systems  engineering  done  before  development  started,  risks 
were  understood.  The  Air  Force  had  a  sound  business  case  that  reflected 
a  realistic  understanding  of  the  challenge  facing  the  SDB  I  development 
program.  The  program’s  total  development  cost  decreased  by  almost  4 
percent  and  the  system  was  available  for  use  1  month  earlier  than  initially 
estimated. 

Our  analysis  of  the  JLTV  requirements  challenge  and  the  ensuing 
systems  engineering  effort  is  summarized  in  figure  7. 


Figure  7:  Joint  Light  Tactical  Vehicle 
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Source:  GAO  analysis  of  Department  of  Defense  data  |  GAO-17-77 


At  the  start  of  product  development  in  2012,  the  Army  and  Marine  Corps 
and  the  JLTV  prime  contractor  had  performed  detailed  systems 
engineering  to  understand  the  challenge  posed  by  requirements  and 
remaining  risks.  The  JLTV  was  expected  to  be  a  family  of  vehicles  built 
on  a  common  automotive  vehicle  platform  to  replace  the  High  Mobility 
Multipurpose  Wheeled  Vehicle  for  some  missions.  In  response  to 
concerns  raised  by  DOD’s  acquisition  executive  regarding  technical 
maturity,  shifting  requirements,  and  affordability,  and  in  light  of  available 
resources,  the  Army  and  Marine  Corps  worked  with  three  contractors 
prior  to  the  start  of  product  development  to  refine  requirements.  The  Army 
and  Marine  Corps  planned  an  incremental  acquisition  approach  using 
mature  technology  and  based  on  a  design  derived  from  prototypes 
developed  and  tested  by  the  winning  contractor  prior  to  the  start  of 
product  development.  Basic  capability  would  be  provided  initially,  with 
enhanced  force  protection,  increased  fuel  efficiency,  greater  payload,  and 
other  improvements  to  be  added  in  later  increments.  As  a  result,  the 
prime  contractor  was  able  to  identify  an  allocated  baseline  and  provide  a 
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mature  product  based  on  a  demonstrated  design  at  the  start  of 
development.  With  the  bulk  of  the  systems  engineering  done  before 
development  started,  the  JLTV  program  had  taken  steps  to  reduce  risk 
and  enable  a  sound  business  case  that  reflected  a  realistic  understanding 
of  the  challenge  it  faced.  The  program  has  decreased  development  costs 
by  6  percent.  Although  acquisition  cycle  time  has  increased  19  months 
from  initial  estimates,  program  officials  noted  that  the  schedule  growth 
was  due  to  issues  unrelated  to  systems  engineering  or  program 
requirements,  specifically  production  contract  bid  protests  that  delayed 
operational  testing  and  the  need  to  add  time  for  Army  users  to  complete 
critical  training. 


Three  Programs  Had 
Some  Requirements 
Challenges  but  Conducted 
Systems  Engineering  to 
Support  Risk  Mitigation 


The  Global  Positioning  System  III  (GPS  III),  Paladin  Integrated 
Management  (PIM),  and  P-8A  Poseidon  Multi-Mission  Maritime  Aircraft 
(P-8A)  programs  provide  examples  in  which  top-level  requirements 
generally  posed  moderate  challenges  with  some  risks  that  were  identified 
and  generally  mitigated  through  systems  engineering.  In  these  cases,  the 
government  and  the  prime  contractor  conducted  some  detailed  systems 
engineering  to  inform  their  program  business  cases  before  starting 
product  development  but  not  all  requirements  risks  were  mitigated,  which 
contributed  to  problems  and  some  cost  and  schedule  growth. 


Global  Positioning  System  III  Our  analysis  of  the  GPS  III  requirements  challenge  and  the  ensuing 

systems  engineering  effort  is  summarized  in  figure  8. 


Figure  8:  Global  Positioning  System  III 
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Although  the  Air  Force  and  GPS  III  prime  contractor  conducted  some 
detailed  systems  engineering  to  inform  the  program’s  business  case 
before  the  start  of  development  in  May  2008,  additional  risks  to  fielding 
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the  capability  existed  due  to  interdependencies  associated  with  a  critical 
ground  control  system  and  military  user  equipment  that  were  being 
developed  and  managed  as  separate  programs.  The  GPS  III  program  is 
expected  to  develop  and  field  a  new  generation  of  satellites  to 
supplement  and  eventually  replace  currently  operational  GPS  satellites. 
As  the  space  segment  of  DOD’s  effort  to  sustain  and  modernize  the  GPS 
system,  the  GPS  III  program  is  one  part  of  a  system  of  systems,  which 
also  includes  the  Next  Generation  Operational  Control  System  (OCX), 
intended  to  replace  the  existing  GPS  ground  system  and  to  operate  both 
current  and  future  satellites,  and  the  Military  GPS  User  Equipment 
(MGUE),  intended  to  provide  the  military  services  with  new  GPS 
receivers.  In  order  to  avoid  problems  associated  with  a  previous  GPS 
satellite  program,  the  Air  Force  planned  an  incremental  acquisition 
approach  to  GPS  III,  using  technologies  that  were  assessed  as  mature 
and  performed  some  systems  engineering  to  gain  an  early  understanding 
of  the  challenge  posed  by  requirements.  According  to  contractor  officials, 
this  systems  engineering,  conducted  during  the  competitive  requirements 
development  phase,  helped  them  better  understand  design  requirements. 
Regardless  of  these  efforts,  the  program  was  negatively  impacted  by 
dependency  on  other  systems. 

Design  and  manufacturing  problems  with  a  key  GPS  III  navigation 
subsystem  delayed  the  program  by  almost  2  years;  however,  the 
satellites’  dependence  on  the  much  delayed  OCX  proved  to  be  a  greater 
challenge  to  sustaining  and  modernizing  the  GPS  system.  The  delivery  of 
the  OCX  Block  1 ,  which  is  required  to  operate  the  GPS  III  satellites  with 
both  legacy  and  new  capabilities,  has  been  delayed  by  6  years  to  July 
2021  and  may  be  delayed  further  because  of  ongoing  developmental 
issues.  As  a  result  of  the  OCX  program’s  delays,  the  Air  Force  is  pursuing 
a  smaller  scale  program  to  modify  the  existing  GPS  operational  control 
system  to  enable  the  operational  use  of  GPS  III  satellites  for  all  legacy 
GPS  functions  until  delivery  of  the  OCX  Block  1 ,  which  will  then  permit 
the  operational  testing  and  use  of  the  GPS  III  satellites’  new  capabilities. 
Due  to  OCX  delays  and  the  likely  timing  of  the  contingency  operations 
system  the  Air  Force  may  need  to  delay  the  launch  of  multiple  GPS  III 
satellites  or  launch  several  without  fully  testing  them.  Moreover,  although 
testing  the  satellites’  functionality  is  not  dependent  on  new  user 
equipment,  timing  of  MGUE  capability  delivery  will  further  postpone — by 
about  a  decade — the  warfighter’s  ability  to  take  advantage  of  the 
upgraded  system’s  new  military  code,  which  offers  greater  resistance  to 
jamming. 
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Paladin  Integrated 
Management 


Our  analysis  of  the  PIM  requirements  challenge  and  the  ensuing  systems 
engineering  effort  is  summarized  in  figure  9. 


Figure  9:  Paladin  Integrated  Management 
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The  Army’s  Paladin  Integrated  Management  (PIM)  program — officially  the 
M109A7  Family  of  Vehicles — began  formal  product  development  as  a 
major  defense  acquisition  program  in  June  2011.  12  The  Army  began 
early  systems  engineering  and  development  work  in  2007.  At  that  time, 
PIM  was  a  service  life  extension  program  for  the  existing  Paladin  M109A6 
and  according  to  program  officials,  was  expected  to  be  a  relatively  minor 
upgrade  to  update  the  fire  control  system  and  replace  the  power  pack  of 
the  existing  system.  The  Army  expected  these  requirements  to  neither 
result  in  a  significant  or  complicated  design  change,  nor  be  costly.  As  part 
of  the  service  life  extension  program,  the  Army  had  identified  both 
functional  and  allocated  baselines.  By  2011,  the  Army  had  made  several 
changes  to  the  program,  with  significant  implications  for  the  design  and 
cost  of  PIM.  Specifically,  the  Army  increased  force  protection  and 
survivability  requirements,  as  directed  by  DOD,  and  added  a  cannon  that 
had  been  in  development  as  part  of  the  Army’s  Future  Combat  Systems 
program  before  it  was  cancelled  in  2009.  These  changes  essentially 
resulted  in  a  new  design  versus  an  upgrade,  and  the  associated  costs 
increased  enough  to  make  PIM  a  major  defense  acquisition  program. 


12  Major  defense  acquisition  programs  are  those  identified  by  DOD  with  a  dollar  value  for 
all  increments  estimated  to  require  eventual  total  expenditure  for  research,  development, 
test,  and  evaluation  of  more  than  $480  million,  or  for  procurement  of  more  than  $2.79 
billion,  in  fiscal  year  2014  constant  dollars.  See  DOD  Instruction  5000.02,  Operation  of 
Defense  Acquisition  System,  enclosure  1,  table  1  (Jan.  7,  2015). 
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P-8A  Poseidon  Multi-Mission 
Maritime  Aircraft  Increment  I 


The  program’s  functional  and  allocated  baselines  were  re-established  in 
2012,  reflecting  both  the  design  changes  and  the  systems  engineering 
done  since  2007,  and  ensured  cost  and  schedule  estimates  were 
sufficiently  informed  for  a  sound  business  case.  Since  201 1 ,  development 
cost  has  increased  5  percent  and  the  program  has  experienced  a 
schedule  delay  of  2  months.  While  not  a  textbook  example  of  how  to  keep 
requirements  stable  during  a  program,  PIM’s  experience  shows  that 
changes  can  be  accommodated  when  accompanied  by  a  suitable 
systems  engineering  effort. 

Our  analysis  of  the  P-8A  Increment  I  requirements  challenge  and  the 
ensuing  systems  engineering  effort  is  summarized  in  figure  10. 


Figure  10:  P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  Increment  I 
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At  the  start  of  P-8A  development  in  2004,  the  Navy  and  the  prime 
contractor  had  completed  some  detailed  systems  engineering  but  an 
allocated  baseline  was  not  yet  established.  The  P-8A  is  expected  to 
replace  an  existing  system,  the  P-3C  Orion,  and  meet  aspects  of  the 
Navy’s  anti-submarine  warfare,  anti-surface  warfare,  and  intelligence, 
surveillance,  and  reconnaissance  capability  requirements.  The  Navy 
planned  an  incremental,  evolutionary  acquisition  approach  focused  on 
providing  a  first  increment  of  capability  to  users  in  the  quickest  and  most 
cost  efficient  manner.  The  Navy  also  expected  P-8A  Increment  I  to  be 
built  on  open  systems  architecture  to  allow  subsequent  increments  to 
more  easily  deliver  increased  capabilities.13  In  addition,  the  airframe  was 

13  An  open  system  allows  system  components  to  be  added,  removed,  modified,  replaced, 
or  sustained  by  consumers  or  different  manufacturers  in  addition  to  the  manufacturer  that 
developed  the  system.  It  also  allows  independent  suppliers  to  build  components  that  can 
plug  into  the  existing  system  through  the  open  connections. 
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to  be  derived  from  an  existing  commercial  aircraft.  According  to  prime 
contractor  officials,  the  P-8A  airframe  design  was  about  50  percent 
common  with  the  commercial  airframe  from  which  it  was  derived.  At  the 
start  of  development,  none  of  P-8A  Increment  I’s  four  critical  technologies 
were  mature,  increasing  risk  associated  with  the  program’s  business 
case.  However,  as  part  of  its  systems  engineering  analysis,  the  Navy  and 
the  prime  contractor  identified  mature  backup  technologies  that  could  be 
used  if  the  critical  technologies  did  not  mature  as  expected  thus  mitigating 
some  of  the  technology  risk.  P-8A  Increment  I  experienced  some  cost 
and  schedule  growth.  That  growth  is  largely  due  to  delays  in  the 
completion  of  the  final  system  design.  As  of  February  2016,  the  program’s 
development  cost  estimate  had  increased  14  percent,  and  its  acquisition 
cycle  time  had  increased  by  4  months. 


Three  Programs  with 
Problematic  Outcomes 
Had  Challenging 
Requirements  and  Limited 
Systems  Engineering 
before  Development  Start 


The  F-35  Lightning  II  (F-35)  and  CH-53K  Heavy  Lift  Replacement 
Helicopter  (CH-53K)  did  not  conduct  adequate  systems  engineering  prior 
to  starting  product  development,  and  as  result  began  development  with 
significant  risks  and  a  limited  understanding  of  the  challenge  posed  by 
their  requirements.  Neither  program  had  established  a  functional  or 
allocated  baseline  before  development  began.  In  fact,  their  allocated 
baselines  were  not  established  until  years  into  product  development.  The 
government  and  prime  contractor  for  the  Integrated  Air  and  Missile 
Defense  (IAMD)  program  did  some  systems  engineering  before  the  start 
of  product  development,  but  the  Army  made  high-level  changes  during 
the  first  year  of  development  to  integrate  additional  systems, 
necessitating  additional  detailed  systems  engineering  work  after  the  start 
of  development. 
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F-35  Lightning 


Our  analysis  of  the  F-35  requirements  challenge  and  the  ensuing  systems 
engineering  effort  is  summarized  in  figure  11. 


Figure  11:  F-35  Lightning  II 
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At  the  start  of  development  in  2001 ,  neither  DOD  nor  the  F-35  prime 
contractor  had  conducted  detailed  systems  engineering  to  adequately 
retire  risk,  establish  an  allocated  baseline,  and  truly  understand  the 
challenge  posed  by  their  requirements.  As  a  result,  the  F-35  program  did 
not  have  a  sound,  executable  business  case.  The  F-35  was  a  next 
generation,  highly  complex  stealth  fighter  with  integrated  avionics  and 
software  intensive  mission  systems  being  developed  in  three  variants  to 
meet  the  needs  of  three  U.S.  military  services  and  various  international 
partners.  When  development  began,  DOD  planned  to  achieve  full 
capability  using  a  single-step  acquisition  approach  and  the  F-35’s  critical 
technologies  were  immature.  DOD’s  initial  program  plans  showed  that 
key  systems  engineering  analyses  to  support  an  allocated  baseline  would 
not  be  complete  until  the  program  was  years  into  development. 

The  bulk  of  F-35  systems  engineering  was  done  after  development  began 
and  the  program  experienced  significant  cost  and  schedule  growth — 
development  costs  increased  60  percent  over  initial  estimates  and  initial 
operational  capability  was  delayed  over  5  years — and  was  restructured 
three  times.  The  first  restructuring  was  driven  by  the  discovery  of 
significant  weight  issues  with  the  F-35B  variant  as  the  program  developed 
its  preliminary  design,  resulting  in  costly  design  changes.  DOD  and  prime 
contractor  data  indicate  that  the  system’s  functional  baseline — detailing 
the  system-level  design — was  not  finalized  until  4  years  into  development. 
As  systems  engineering  continued  and  testing  began,  additional 
discoveries  were  made,  requiring  further  design  changes  and  contributing 
to  the  two  additional  restructurings.  Ultimately,  the  program  realized  that 
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CH-53K  Heavy  Lift 
Replacement  Helicopter 


some  capabilities  originally  envisioned  when  development  began  would 
not  work  as  expected  and  would  have  to  be  completed  as  part  of  follow- 
on  development  program. 

Our  analysis  of  the  CH-53K  requirements  challenge  and  the  ensuing 
systems  engineering  effort  is  summarized  in  figure  12. 


Figure  12:  CH-53K  Heavy  Lift  Replacement  Helicopter 


1  Factors  I 

Acquisition  approach 

Single  step 

Technology  status 

Immature 

Design  maturity 

New 

Interdependency 

Limited 

Development  Functional  Allocated 

start  baseline  baseline 

(12/05)  (6/06)  (6/07) 


Source:  GAO  analysis  of  Department  of  Defense  data.  |  GAO-17-77 


At  the  start  of  development  in  2005,  neither  the  Marine  Corps  nor  the  CH- 
53K  prime  contractor  had  conducted  detailed  systems  engineering  to 
adequately  retire  risk,  establish  an  allocated  baseline,  and  fully 
understand  the  challenge  posed  by  their  requirements.  They  had 
established  neither  a  performance  specification  nor  a  functional  baseline. 
As  a  result,  the  CH-53K  program  did  not  have  a  sound,  executable 
business  case.  The  Marine  Corps  and  the  prime  contractor  disagreed  on 
what  specific  systems  engineering  tasks  were  needed,  and  initial  plans 
showed  that  key  analyses  would  not  be  complete  until  the  system  was 
well  into  development.  The  Marine  Corps  expected  the  CH-53K  to  provide 
improvements  relative  to  the  CH-53E  in  range  and  payload,  survivability 
and  force  protection,  reliability  and  maintainability,  coordination  with  other 
assets,  and  overall  cost  of  ownership.  As  a  result  and  in  order  to  meet 
requirements,  an  entirely  new  aircraft  design  was  needed.  According  to 
the  prime  contractor,  none  of  the  parts  in  the  new  design  were  common 
parts  with  the  CH-53E.  When  development  began,  the  Marine  Corps 
planned  to  achieve  full  capability  using  a  single-step  acquisition 
approach.  At  that  same  time,  the  system’s  critical  technologies  were  not 
mature. 

Nearly  all  CH-53K  systems  engineering  was  done  after  development 
began,  and  the  program  experienced  significant  cost  and  schedule 
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Integrated  Air  and  Missile 
Defense 


growth.  Because  the  program  office  and  prime  contractor  disagreed  on 
what  systems  engineering  tasks  needed  to  be  accomplished,  problems 
began  immediately  following  the  start  of  development.  Once  systems 
engineering  began  and  problems  were  identified,  the  Marine  Corps  chose 
to  defer  some  key  performance  capabilities  to  reduce  development  costs, 
and,  in  2010,  the  program  was  restructured  with  a  new  baseline  approved 
in  2013.  Discoveries  during  ground  testing  and  qualification  drove 
additional  unanticipated  design  changes  and  delays.  As  a  result,  the 
program’s  initial  production  decision  was  delayed  8  months,  and  the 
program  now  plans  to  establish  a  new  cost  baseline.  In  total,  the 
program’s  development  costs  have  increased  51  percent  over  initial 
estimates,  and  its  initial  operational  capability  has  been  delayed  by  over  4 
years. 

Our  analysis  of  the  IAMD  requirements  challenge  and  the  ensuing 
systems  engineering  effort  is  summarized  in  figure  13. 


Figure  13:  Integrated  Air  and  Missile  Defense 
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Although  the  Army  and  IAMD  prime  contractor  had  conducted  some 
detailed  systems  engineering  and  technologies  were  assessed  as  nearly 
mature  at  the  start  of  product  development,  the  IAMD  program’s  business 
case  was  critically  dependent  on  sensors  and  weapons  that  had  been 
developed  and  managed  as  separate  programs.  IAMD  is  expected  to 
network  sensors,  weapons,  and  a  common  battle  command  system 
across  an  integrated  fire  control  network  to  support  the  engagement  of  air 
and  missile  threats.  The  Army  planned  an  incremental  acquisition 
approach  and  held  a  competition  for  preliminary  design  to  mitigate  risk  in 
May  2009.  Prime  contractor  officials  told  us  they  conducted  systems 
engineering  prior  to  the  start  of  development,  which  may  have  helped 
them  better  understand  design  requirements.  While  a  functional  baseline 
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was  established  prior  to  the  start  of  product  development  and  the  prime 
contractor  built  a  basic  design  prototype  to  demonstrate  for  the 
government  that  the  design  architecture  would  work,  the  system’s 
allocated  baseline  was  not  fully  established  until  after  the  start  of 
development. 

System  interdependency  has  posed  challenges  to  the  IAMD  program  in 
several  ways.  The  Army  made  high-level  changes  in  IAMD  during  the  first 
year,  which  resulted  in  the  program  having  to  integrate  additional  systems 
and  causing  significant  changes  in  how  the  systems  were  to  interact. 
According  to  prime  contractor  officials,  three  of  the  four  IAMD  critical 
technologies  rely  on  interfacing  with  other  systems  and  many  of  these 
interfaces  were  not  defined  until  after  the  development  contract  was 
awarded.  IAMD  has  encountered  challenges  integrating  new  software 
from  the  contractor  with  other  acquisition  programs  critical  to  its 
functionality.  IAMD  development  cost  has  increased  over  60  percent 
since  the  start  of  development,  with  about  half  of  the  increase  occurring  in 
the  first  year.  The  program  has  been  rebaselined  and  initial  operational 
capability  has  been  deferred  by  2  years.  This  experience  is  somewhat 
similar  to  that  of  the  GPS  III  with  an  important  distinction.  While 
challenges  with  GPS  Ill’s  interdependence  with  other  systems  did  not 
significantly  affect  the  design  of  the  satellites  themselves,  the 
interdependence  of  IAMD  did  have  significant  implications  for  its  software 
design,  which  were  not  recognized  until  after  the  start  of  product 
development. 


Knowledge  of 
Systems  Engineering 
Can  Enhance 
Oversight  before  a 
Program  Starts 
Product  Development 


Systems  engineering,  if  done  well,  can  be  a  key  enabler  for  establishing  a 
sound  business  case  for  a  program  before  the  start  of  product 
development.  By  the  same  token,  it  can  provide  indications  to  Congress 
and  other  organizations  responsible  for  oversight  as  to  the  soundness  of 
DOD’s  approach  to  undertaking  a  new  program.  The  decision  to  begin 
product  development  is  the  last  opportunity  in  the  acquisition  process — 
prior  to  committing  to  a  major  financial  investment — to  level  a  product’s 
requirements  to  available  resources,  thereby  establishing  a  good 
business  case.  A  sound  business  case  provides  credible  evidence  that 
(1)  the  warfighter’s  needs  are  valid  and  that  they  can  best  be  met  with  the 
chosen  concept;  and  (2)  the  chosen  concept  can  be  developed  and 
produced  within  existing  resources — that  is,  proven  technologies, 
sufficient  design  knowledge,  adequate  funding,  and  adequate  time  exist 
to  deliver  the  product  when  it  is  needed.  A  program  should  not  go  forward 
into  product  development,  nor  should  this  step  be  funded,  unless  a  sound 
business  case  can  be  made.  Therefore,  Congress,  which  is  responsible 
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for  making  that  funding  decision,  should  also  have  insight  into  the 
soundness  of  the  business  case.  However,  given  current  budget 
processes  and  mechanics  in  DOD  today,  these  congressional  decisions 
must  be  made  well  in  advance  of  the  start  of  product  development  and 
establishment  of  the  final  business  case. 

Systems  engineering  is  essential  to  establishing  a  sound  business  case. 
We  have  previously  found  that  key  enablers  of  a  good  business  case 
include  firm,  feasible  requirements,  mature  technology,  an  incremental, 
knowledge-based  acquisition  strategy,  and  a  realistic  cost  estimate.  Our 
work  examining  commercial  companies’  best  practices  has  found  that 
requirements  should  be  clearly  defined,  affordable,  and  clearly 
informed — thus  tempered — by  systems  engineering  prior  to  the  start  of 
product  development.  Once  programs  begin,  requirements  should  not 
change  without  assessing  their  potential  disruption  to  the  program.  In 
addition,  science  and  technology  organizations  should  shoulder  the 
technology  development  burden,  proving  technologies  can  work  as 
intended  before  they  are  included  in  a  weapon  system  program.  The 
principle  is  not  to  avoid  technical  risk,  but  rather  take  risk  early  and 
resolve  it  ahead  of  the  start  of  product  development.  Programs  should 
use  an  incremental,  knowledge-based  acquisition  strategy.  We  have 
found  that  rigorous  systems  engineering  coupled  with  more  achievable 
requirements  are  essential  to  achieve  faster  delivery  of  needed  capability 
to  the  warfighter.  Building  on  mature  technologies,  such  a  strategy 
provides  time,  money,  and  other  resources  for  a  stable  design,  building 
and  testing  of  prototypes,  and  demonstration  of  mature  production 
processes.  Finally,  a  good  business  case  will  have  a  realistic  cost 
estimate  based  on  a  knowledge-based  acquisition  strategy,  independent 
assessments,  and  sound  methodologies. 

Over  the  years,  and  as  previously  noted  in  this  report,  we  have  found  that 
for  a  program  to  deliver  a  successful  product  with  available  resources, 
high  levels  of  knowledge — informed  by  systems  engineering — must  be 
demonstrated  before  significant  commitments  are  made.14  This  requires 
the  user  and  developer  to  negotiate  whatever  trade-offs  are  needed  to 
achieve  a  match  between  the  user’s  requirements  and  the  developer’s 
resources  before  product  development  begins.  While  it  would  seem  that 
taking  such  an  approach  would  be  axiomatic,  we  have  found  that  it  is 


14GAO-1 5-469,  GAO-12-400SP,  GAO-09-326SP,  GAO-02-701,  and  GAO-01-288. 
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not.15  On  the  contrary,  we  have  previously  found  that  there  are  strong 
incentives  within  the  culture  of  weapon  system  acquisition  to  overpromise 
a  prospective  weapon’s  performance  while  understating  its  likely  cost  and 
schedule  demands.16  Competition  with  other  programs  vying  for  defense 
dollars  puts  pressure  on  program  sponsors  to  project  unprecedented 
levels  of  performance  (often  by  counting  on  unproven  technologies)  while 
promising  low  cost  and  short  schedules.  These  incentives,  coupled  with  a 
marketplace  that  is  characterized  by  a  single  buyer  (DOD),  low  volume, 
and  limited  number  of  major  sources,  create  a  culture  in  weapon  system 
acquisition  that  encourages  undue  optimism  about  program  risks  and 
costs. 

A  particular  challenge  for  Congress  is  the  fact  that  committees  must  often 
consider  requests  to  authorize  and  fund  a  new  program  well  ahead  of  the 
start  of  product  development — the  point  at  which  key  business  case 
information  would  be  presented.  For  example,  if  DOD  has  scheduled  a 
decision  to  start  a  new  development  program  in  August  2017,  the  funding 
needed  for  the  first  year  of  development  would  have  to  be  included  in 
DOD’s  fiscal  year  2017  budget  request.  This  budget  request  would  be 
submitted  to  Congress  in  February  2016,  or  18  months  before  the  actual 
program  decision.  Given  this  time  lag,  Congress  could  be  making  critical 
funding  decisions — which  in  effect  authorize  the  start  of  development — 
with  limited  information  about  program  risk  factors,  systems  engineering 
progress,  and  the  soundness  of  the  program’s  business  case. 

The  nine  case  studies  we  examined  for  this  report  suggest  that 
understanding  the  dynamic  between  program  requirements,  risks,  and  the 
requisite  systems  engineering  analysis  can  facilitate  early  oversight. 
Specifically,  when  the  top-level  requirements  for  what  could  be  a  new 
major  weapon  system  are  initially  identified  in  a  draft  Capability 
Development  Document,  the  challenge  those  requirements  pose  and  how 
it  relates  to  the  four  factors  we  identified — acquisition  approach, 
technology  status,  design  maturity,  and  system  interdependence — can  be 
observed.  Once  the  challenge  is  framed,  the  systems  engineering  plan 


15  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs, 
GAO-16-329SP  (Washington,  D.C.:  Mar.  31, 2016);  Defense  Acquisitions:  Assessments 
of  Selected  Weapons  Programs,  GAO-15-342SP  (Washington,  D.C.:  Mar.  12,  2015);  and 
Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and  Schedule 
Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13,  2006). 

16  GAO,  Defense  Acquisitions:  Joint  Action  Needed  by  DOD  and  Congress  to  Improve 
Outcomes,  GAO-1 6-1 87T  (Washington,  D.C.:  Oct.  27,  2015). 
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can  be  refined  to  obtain  the  requisite  knowledge  to  (1)  identify  expectation 
gaps  and  other  risks,  and  (2)  resolve  them  by  the  start  of  product 
development.  Ultimately,  the  final  Capability  Development  Document 
could  then  be  informed  by  an  allocated  baseline  (the  preliminary  design) 
and  provide  a  sound  basis  for  an  executable  business  case.  This 
sequence  of  events  is  notionally  depicted  in  figure  14  along  with  when  a 
budget  to  start  a  new  program  is  submitted  to  Congress. 


Figure  14:  Key  Systems  Engineering  Stages  Needed  for  Sound  Business  Case 


Budget  request 
submitted  to 
Congress 


Initial 

requirements 


Understanding 
of  factors 


Functional 

baseline 


Final 

requirements 


* 


Allocated  baseline 
(Preliminary  design) 


Business  case 
for  start  of 
product 
development 


Source:  GAO  analysis  of  Department  of  Defense  data.  |  GAO-17-77 


While  congressional  insight  at  the  time  of  the  funding  decision  is  limited, 
DOD  policy  directs  program  managers  to  provide  certain  documents  to 
DOD  decision  makers  that  when  taken  together  could  provide  a  picture  of 
a  proposed  program’s  risk  factors  and  systems  engineering  status. 
Programs  are  generally  required  to  provide  versions  of  these  documents 
to  DOD  decision  makers  prior  to  each  major  milestone  review,  including 
the  start  of  development.  However,  it  is  not  clear  whether  Congress  is 
given  the  same  information  DOD  decision  makers  are  given  when  DOD 
submits  its  budget  request  for  funding  to  begin  a  new  program.  Those 
documents  include  an  acquisition  strategy,  a  test  and  evaluation  master 
plan,  a  technology  readiness  assessment,  and  a  systems  engineering 
plan. 

The  systems  engineering  plan  is  of  particular  interest  because  it  brings 
the  risk  factors  that  we  have  identified  together  with  a  proposed 
program’s  systems  engineering  knowledge.  According  to  DOD  acquisition 
policy,  the  plan  will  be  submitted  for  approval  for  each  milestone  review, 
beginning  with  the  milestone  review  to  approve  the  start  of  the  technology 
maturation  and  risk  reduction  phase — the  acquisition  phase  that  precedes 
the  start  of  product  development.  DOD  views  the  plan  as  a  “living 
document”  that  it  expects  to  be  updated  as  needed  throughout  the 
acquisition  process.  The  plan  is  expected  to  support  the  acquisition 
strategy,  including  the  program  interdependencies,  and  communicate  the 
overall  technical  approach  to  balance  system  performance,  life-cycle  cost, 
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and  risk  in  addressing  warfighter  needs.  It  should  describe  the  program’s 
overall  technical  approach,  including  key  technical  risks,  processes, 
resources,  organization,  metrics,  and  design  considerations.  It  should 
also  detail  the  timing  and  criteria  for  conducting  systems  engineering 
technical  reviews,  including  those  that  will  establish  the  functional, 
allocated,  and  product  level  baselines.  In  addition,  the  plan  should 
address  system  integration  risks,  including  risks  related  to  external 
dependencies  which  are  outside  the  program  manager’s  span  of  control 
in  order  to  ensure  timely  design,  development,  deployment,  and 
sustainment  of  the  system.  Key  aspects  of  the  plan  are  intended  to 
support  more  detailed  technical  planning  in  order  to  provide  effective 
management  and  control  of  the  program’s  technical  progress  and  the 
execution  of  risk  mitigation  activities. 

If  the  key  factors  that  frame  the  requirements  challenge  are  understood,  a 
functional  baseline  is  established,  and  a  realistic  plan  exists  to  complete 
the  allocated  baseline,  Congress  will  have  more  assurance  that  a 
program  is  on  a  path  to  a  sound  business  case — informed  by  an  allocated 
baseline — as  it  considers  funding  requests  for  development  start,  and  a 
systems  engineering  plan  could  help  to  pull  these  elements  together. 


Conclusions 


Weapon  system  development  programs  involve  unknowns  that  translate 
into  cost  and  schedule  risk.  This  is  the  nature  of  any  product  development 
program.  The  case  studies  in  this  report  illustrate  a  strong  relationship 
between  requirements,  systems  engineering,  and  program  outcomes. 
They  show  that  one  key  to  avoid  poor  outcomes  in  major  weapon  system 
development  programs  is  not  necessarily  to  eliminate  all  risks,  but  instead 
to  invest  in  detailed  systems  engineering  analysis  to  understand  the 
design  implications  of  requirements  and  reconcile  them  with  the 
resources  that  can  be  reasonably  expected.  At  a  minimum,  this 
reconciliation,  or  match,  should  be  evidenced  by  an  allocated  baseline — 
that  is,  a  preliminary  design — before  committing  to  a  product  development 
program.  Our  cases  indicate  that  regardless  of  the  challenge  posed  by 
initial  requirements,  systems  engineering,  if  done  well,  can  be  a  key 
enabler  of  a  sound  business  case  for  starting  a  new  development 
program.  This  helps  to  ensure  that  if  a  program  takes  risks,  those  risks 
are  clearly  identified,  and  any  resource  consequences  are  acknowledged 
and  accounted  for  in  cost  and  schedule  estimates.  In  some  cases — as 
seen  with  JLTV  and  PIM — this  process  could  take  time  and  result  in 
requirements  changes,  but,  if  done  before  committing  to  a  product 
development,  it  is  more  likely  to  result  in  a  sound  and  executable 
business  case. 
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Understanding  the  dynamic  between  program  requirements,  risks,  and 
systems  engineering  can  facilitate  early  oversight — thus  offering  a 
potential  curb  to  overpromising.  When  the  top-level  requirements  for  what 
could  be  a  new  major  weapon  system  are  initially  identified  in  a  draft 
Capability  Development  Document,  DOD  should  clearly  understand  the 
challenge  presented  by  those  requirements  as  framed  by  the  four  factors 
we  identified.  Having  a  clear  understanding  of  the  challenge  presented  by 
the  initial  requirements,  coupled  with  a  systems  engineering  plan  laid  out 
against  a  schedule  to  achieve  a  sound,  executable  business  case  before 
the  start  of  product  development  would  provide  Congress  and  other 
oversight  organizations  a  means  to  assess  the  soundness  of  a  proposed 
acquisition  program.  Importantly,  it  would  be  possible  to  assess  progress 
made  by  the  program  against  the  plan,  using  the  identification  of  key 
systems  engineering  baselines  as  waypoints.  As  progress  is  made, 
artifacts  of  knowledge — in  terms  of  the  systems  engineering  baselines  as 
well  as  decisions  made  to  trade  off  requirements,  choose  alternate  design 
or  technology  solutions,  and  to  add  time  and  money  to  close  gaps — 
should  also  be  available.  While  Congress  would  still  likely  be  in  the 
position  of  considering  a  budget  request  to  start  a  program  well  before 
DOD’s  actual  decision  to  start  product  development,  the  systems 
engineering  plan  and  progress  made  against  it  would  provide  more  robust 
input  to  Congress’s  deliberations.  This  could  also  provide  Congress  with 
better  insight  into  the  risks  facing  a  proposed  development  program — 
including  the  risks  a  program  may  be  taking  after  the  start  of  product 
development  in  the  event  that  the  desired  level  of  systems  engineering  is 
not  complete. 


Matter  for 

Congressional 

Consideration 


To  enhance  program  oversight  and  provide  more  robust  input  to  budget 
deliberations,  Congress  should  consider  requiring  DOD  to  report  on  each 
major  acquisition  program’s  systems  engineering  status  in  the 
department’s  annual  budget  request,  beginning  with  the  budget 
requesting  funds  to  start  development.  The  information  could  be 
presented  on  a  simple  timeline — as  done  for  the  case  studies  in  this 
report — and  at  a  minimum  should  reflect  the  status  of  a  program’s 
functional  and  allocated  baselines  as  contained  in  the  most  current 
version  of  the  program’s  systems  engineering  plan. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  DOD  for  review  and  comment.  DOD’s 
written  comments  are  reprinted  in  appendix  II  of  this  report.  That  draft 
report  included  a  recommendation  that  DOD  submit  the  systems 
engineering  plans  of  each  new  proposed  development  program  to 
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Congress  at  the  same  time  the  budget  requesting  funds  to  begin 
development  is  sent  to  Congress.  DOD  did  not  agree  with  our 
recommendation.  In  its  comments,  DOD  agreed  that  early  systems 
engineering  reduces  risk  and  establishes  a  solid  foundation  for  program 
success.  However,  DOD  noted  that  the  systems  engineering  plan  is  a 
management  tool  to  guide  a  program’s  system  engineering  activities  and 
the  timing  of  the  plan  and  any  updates  are  not  aligned  to  inform  a  budget 
decision  that  could  occur  as  much  as  18  months  prior  to  program 
initiation.  DOD  also  noted  that  it  believes  that  existing  statutory 
certifications  and  reports  submitted  to  Congress  contain  adequate 
information  regarding  program  risk  and  technical  maturity.  We  appreciate 
DOD’s  thoughtful  response  and  agree  that  existing  statutory  certifications 
and  reports  provide  some  level  of  insight  into  program  risks  and  general 
assurance  that  plans  are  in  place  to  address  those  risks.  However,  it  is 
not  clear  whether  they  provide  adequate  detail  about  progress  against 
those  plans,  such  as  whether  a  functional  baseline  has  been  established 
or  when  an  allocated  baseline  will  be  established.  In  addition,  those 
certifications  and  reports  are  linked  to  program  milestone  reviews  and  are 
not  specifically  aligned  to  inform  program  budget  decisions.  Documents 
such  as  the  systems  engineering  plan  are  “living  documents”  that  are 
updated  as  needed  throughout  the  acquisition  process  and  could  be 
made  available  to  inform  the  budget  process.  Therefore,  we  continue  to 
believe  that  linking  robust  insight  into  systems  engineering  progress,  like 
the  information  contained  in  the  systems  engineering  plan,  with  the  timing 
of  congressional  budget  deliberations  would  be  beneficial  and  as  such 
are  now  including  a  matter  for  congressional  consideration.  Given  that  the 
systems  engineering  plan  is  updated  during  the  acquisition  process, 
linking  systems  engineering  progress  to  budget  requests  need  not  be 
onerous.  As  noted  above,  this  could  be  accomplished  using  a  top-level 
timeline  of  less  than  a  page,  as  done  in  the  case  studies  in  this  report. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees  and  offices;  the  Secretary  of  Defense;  the  Secretaries  of  the 
Army,  Navy,  and  Air  Force.  In  addition,  the  report  will  be  made  available 
at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  has  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4841 .  Contact  points  for  our  offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
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of  this  report.  Staff  members  making  key  contributions  to  this  report  are 
listed  in  appendix  III. 


Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
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The  Honorable  John  McCain 
Chairman 
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United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 
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Chairman 
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Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Objectives,  Scope  and 
Methodology 


This  report  (1)  identifies  a  framework  for  assessing  the  challenges  and 
risks  associated  with  program  requirements  and  the  extent  of  systems 
engineering  done  before  product  development  begins;  (2)  illustrates  the 
relationship  between  systems  engineering  and  program  outcomes;  and 
(3)  assesses  implications  for  program  management  and  oversight. 

To  conduct  our  work,  we  selected  a  non-generalizable  sample  of  9  major 
defense  acquisition  programs,  out  of  a  total  portfolio  of  79  programs,  to 
include  at  least  2  from  each  service  and  Department  of  Defense  (DOD) 
wide  programs.  The  specific  case  study  programs  we  selected  are 
identified  in  table  1 . 


Table  3:  Case  Study  Programs 

Lead  Service 

Program 

Description 

Air  Force 

KC-46A  Tanker  Modernization  Program 

Aerial  refueling  tanker  aircraft 

Global  Positioning  System  III 

Satellites 

Small  Diameter  Bomb  Increment  1 

Air-to-ground  precision  munitions 

Army 

Integrated  Air  and  Missile  Defense 

Network  of  sensors  and  weapons  with  a 
common  battle  command  system 

Paladin  Integrated  Management 

Self-propelled  howitzer  and  tracked 
ammunition  carrier 

DOD 

F-35  Lightning  II  Program 

Stealthy,  strike  fighter  aircraft 

Joint  Light  Tactical  Vehicle 

Tactical  wheeled  vehicles 

Navy/Marine  Corps 

CH-53K  Heavy  Lift  Replacement  Helicopter 

Personnel  and  equipment  transport  aircraft 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft 
Increment  1 

Anti-surface,  anti-submarine  and 
intelligence,  surveillance  and 
reconnaissance  aircraft 

Source:  GAO  analysis  of  DOD  data  |  GAO-17-77 


To  assess  the  challenges  and  risks  associated  with  program 
requirements  and  the  extent  of  systems  engineering  done  by  each  of  the 
9  programs  before  development  began,  we  analyzed  top-level  program 
requirements  such  as  key  performance  parameters,  key  system 
attributes,  and  other  system  attributes,  as  well  as  lower-level  derived 
requirements  like  system  design  specifications  and  design  drawings.  For 
each  case  study  program,  we  requested  and  analyzed  the  number  of 
requirements  identified  for  each  baseline  (functional,  allocated,  and 
product  level)  at  the  time  of  system  engineering  reviews  and  program 
milestones  and  compared  the  results  to  DOD  systems  engineering 
guidance,  DOD  acquisition  policy,  and  our  previous  GAO  reports 
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Appendix  I:  Objectives,  Scope  and 
Methodology 


examining  weapons  systems  acquisitions  and  best  practices  for  product 
development.1  To  ensure  our  understanding  of  commercial  best  practices 
remained  current,  we  interviewed  officials  from  leading  companies 
including  Caterpillar,  Inc.,  Cummins,  Inc.,  Proctor  and  Gamble,  and 
Motorola  Solutions.  In  addition,  we  assessed  documents  and  data  from 
system  engineering  reviews  and  program  reported  technology  readiness 
levels;  and  we  reviewed  program  acquisition  strategies,  capability 
development  documents  or  operational  requirements  documents, 
acquisition  baselines,  and  selected  acquisition  reports. 

To  assess  the  relationship  between  systems  engineering  and  program 
outcomes,  we  analyzed  requirements,  cost,  and  schedule  documentation 
for  each  of  our  case  study  programs,  and  then  met  with  relevant  program 
officials  and  prime  contractors  to  obtain  relevant  program  documents, 
data,  and  historical  information.  We  reviewed  selected  acquisition  and 
Defense  Acquisition  Executive  Summary  reports,  budget  data,  DOD 
Systems  Engineering  assessments  of  preliminary  and  critical  design 
reviews,  and  previous  GAO  reports  such  as  our  annual  reviews  of  major 
defense  acquisition  programs  and  program  specific  reports.2 

To  identify  program  oversight  implications,  we  also  reviewed  relevant 
acquisition  statutes,  DOD  acquisition  policy,  systems  engineering 
guidance,  and  previous  GAO  reports  examining  weapons  systems 


1  GAO,  Defense  Acquisition  Process:  Military  Service  Chiefs’  Concerns  Reflect  Need  to 
Better  Define  Requirements  before  Programs  Start,  GAO-15-469  (Washington,  D.C.:  June 
1 1 , 201 5);  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost 
and  Schedule  Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.: 
Apr.  13,  2006);  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early 
Improves  Acquisition  Outcomes.  GAO-02-701  (Washington,  D.C.:  July  15,  2002);  and 
Best  Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better  Weapon 
System  Outcomes,  GAO-01-288  (Washington,  D.C.:  Mar.  8,  2001). 

2  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs 
GAO-16-329SP  (Washington,  D.C.:  Mar.  31, 2016);  F-35  Joint  Strike  Fighter:  Continued 
Oversight  Needed  as  Program  Plans  to  Begin  Development  of  New  Capabilities 
GAO-16-390  (Washington,  D.C.:  Apr.  14,  2016);  KC-46  Tanker  Aircraft:  Challenging 
Testing  and  Delivery  Schedules  Lie  Ahead  GAO-16-346  (Washington,  D.C.:  Apr.  8,  2016); 
Defense  Acquisitions:  CH-53K  Helicopter  Program  Has  Addressed  Early  Difficulties  and 
Adopted  Strategies  to  Address  Future  Risks  GAO-1 1-332  (Washington,  D.C.:  Apr.  4, 

201 1 );  Defense  Acquisitions'.  Issues  to  Be  Considered  as  DOD  Modernizes  Its  Fleet  of 
Tactical  Wheeled  Vehicles  GAO-11-83  (Washington,  D.C.:  Nov.  5,  2010);  Global 
Positioning  System:  Challenges  in  Sustaining  and  Upgrading  Capabilities  Persist 
GAO-10-636  (Washington,  D.C.:  Sept.  15,  2010);  and  Defense  Acquisitions:  Strong 
Leadership  Is  Key  to  Planning  and  Executing  Stable  Weapon  Program  GAO- 10-522 
(Washington,  D.C.:  May  6,  2010). 
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acquisitions  and  best  practices  for  product  development.  To  obtain 
additional  insights  into  these  implications,  we  spoke  with  knowledgeable 
DOD  officials  including  program  managers  and  prime  contractors  for  our 
case  study  programs.  We  assessed  the  reliability  of  DOD  and  contractor 
data  by  reviewing  existing  information  about  the  data  and  interviewing 
agency  officials  knowledgeable  about  the  data.  We  spoke  to  both 
program  and  prime  contractor  officials  for  each  of  the  case  study 
programs  and  determined  that  the  data  were  sufficiently  reliable  for  the 
purposes  of  our  reporting  objectives. 

We  conducted  this  performance  audit  from  June  2015  to  November  2016 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 
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RESEARCH 
AND  ENGINEERING 


ASSISTANT  SECRETARY  OF  DEFENSE 

3030  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1  -3030 


OCT  2  4  2015 


Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  Government  Accountability 
Office  (GAO),  GAO-17-77,  “WEAPON  SYSTEM  REQUIREMENTS:  Detailed  Systems 
Engineering  Prior  to  Product  Development  Positions  Programs  for  Success”  dated  September  23, 
2016  (GAO  Code  100167).  Detailed  comments  on  the  report  recommendations  are  enclosed. 


Sincerely, 


Stephen  P.  Welby 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  DATED  SEPTEMBER  23,  2016 
GAO-17-77  (GAO  CODE  100167) 

“WEAPON  SYSTEM  REQUIREMENTS:  DETAILED  SYSTEMS  ENGINEERING 
PRIOR  TO  PRODUCT  DEVELOPMENT  POSITIONS  PROGRAMS  FOR  SUCCESS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION:  To  ensure  that  more  robust  information  about  the  risks  and  requisite 
systems  engineering  status  is  available  to  Congress  to  facilitate  oversight  and  inform  program 
funding  decisions,  the  GAO  recommends  that  the  Secretary  of  Defense  submit  the  systems 
engineering  plans  of  each  new  proposed  development  program  to  Congress  at  the  same  time  the 
budget  requesting  funds  to  begin  development  is  sent  to  Congress, 

DoD  RESPONSE:  Non-concur.  The  Department  of  Defense  agrees  with  the  GAO’s  overall 
finding  that  early,  disciplined  systems  engineering  reduces  program  risk  and  establishes  a  solid 
foundation  for  success.  However,  the  Department  does  not  agree  that  the  Systems  Engineering 
Plan  (SEP)  is  the  most  effective  means  to  provide  Congress  with  insight  into  the  program’s  risk. 
The  Department  believes  that  current  certification  requirements  provide  a  more  relevant  tool  for 
this  purpose. 

The  SEP  is  a  management  tool  to  guide  a  program’s  systems  engineering  activities.  It  is 
formally  updated  immediately  prior  to  each  milestone  decision  to  focus  upon  the  upcoming 
acquisition  phase.  Currently,  the  timing  of  the  SEP  and  any  updates  are  not  aligned  to  inform  a 
budgeting  request  for  program  initiation.  As  the  report  points  out,  the  budgeting  request  could 
happen  as  early  as  1 8  months  before  the  program  initiation  milestone  decision. 

Current  certification  and  reports  already  submitted  to  Congress  contain  information  regarding 
program  risk  and  technical  maturity.  Congress  recently  amended  the  2366a  requirements  to 
include  a  determination  that  there  is  a  plan  to  reduce  any  identified  areas  of  risk  before  Milestone 
A  approval.  Congress  also  amended  statutory  requirements  for  program  acquisition  strategies, 
requiring  a  comprehensive  approach  to  identifying  and  addressing  risk  against  a  set  of  risk 
factors.  Section  2366b  prohibits  (unless  waived)  major  defense  acquisition  programs  from 
receiving  a  Milestone  B  decision  until  a  Preliminary  Design  Review  has  been  conducted,  and  the 
Milestone  Decision  Authority  (MDA)  considers  the  risk  factors  of  the  acquisition  strategy  and 
certifies  that  the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment. 

The  activities  inherent  in  these  reporting  requirements  are  timely  and  facilitate  Congressional 
visibility  into  program  risks,  aligning  well  with  the  findings  of  the  GAO  report. 
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GAO  Contact 


Michael  J.  Sullivan,  202-512-4841  orsullivanm@gao.gov 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card,  MasterCard, 

Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 

Washington,  DC  20548 

Strategic  Planning  and 
External  Liaison 

James-Christian  Blockwood,  Managing  Director,  spel@gao.gov,  (202)  512-4707 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7814, 
Washington,  DC  20548 
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